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ABSTRACT 

When  thermal  imaging  sensors  are  used  for  automatic  target  detection  and  recognition  of  ground 
targets,  the  performance  is  often  limited  by  the  small  thermal  contrast  between  target  and 
background.  In  comparison,  polarimetric  imaging  is  minimally  dependent  on  the  thermal 
characteristics  of  the  target.  Polarimetric  data  depends,  instead,  on  the  three-dimensional  shape 
and  surface  properties  of  the  target.  Physics  Innovations  Inc.  and  Lockheed  Martin  Corp.  have 
an  ongoing  program  to  develop  polarimetric  imaging  systems  for  automatic  target  detection  and 
recognition.  In  this  paper,  we  will  discuss  two  novel  technologies  which  we  have  recently 
developed  for  these  imaging  systems.  The  first  is  a  two-dimensional  array  of  microscale 
retarders.  This  retarder  array,  when  combined  with  a  two-dimensional  array  of  linear  polarizers 
and  a  focal  plane  array,  will  enable  the  imaging  of  the  four  Stokes  parameters  at  video  frame 
rates  without  the  mechanical  rotation  of  optical  components.  The  retarder  elements  are  highly 
achromatic.  Furthermore,  the  retarders  have  a  large  field  angle.  A  large  field  angle  will 
facilitate  the  use  of  low  f-number  optics  for  greater  sensitivity.  The  second  technology 
developed  for  polarimetric  imaging  systems  are  target  detection  and  recognition  algorithms 
which  optimally  combine  the  Stokes  images. 


1.  POLARIZATION-SENSITIVE  THERMAL  IMAGING 

1.1.  POLARIZED  THERMAL  EMISSION  AND  SURFACE  ORIENTATION 

Conventional  methods  for  passively  detecting  and  identifying  targets  often  rely  on 
sensing  the  intensity  of  infrared  light  reflected  or  emitted  by  objects.  A  fundamental  problem 
associated  with  sensing  the  intensity  is  that  intensity  gives  one  parameter  while  the  orientation  of 
surface  elements  has  two  degrees  of  freedom.  Physics  Innovations  Inc.  has  developed  a  thermal 
imaging  technique  for  determining  surface  orientation  where,  in  each  image  pixel,  two 
parameters  are  sensed  simultaneously.  The  two  parameters,  percent  of  polarization  P  and  angle 
of  the  plane  of  polarization  §  are  directly  related  to  the  two  angles  of  surface  orientation. 
Although  thermal  infrared  intensity  images  of  terrestrial  scenes  have  low  contrast,  images  of  P 
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and  (|)  often  have  high  contrast  for  different  surface  orientations.  This  high  contrast  should 
facilitate  image  segmentation  and  classification  of  objects. 

Polarization  also  gives  useful  information  about  the  surface  properties  of  the  object. 
Man-made  objects  have  unnaturally  smooth  surfaces,  leading  to  radiation  with  greater 
polarization.  Natural  backgrounds  such  as  grass,  trees,  dirt,  and  sand  emit  and  reflect  radiation 
that  is  less  polarized. 

Researchers  have  recognized  the  potential  usefulness  of  imaging  polarization  data  for 
target  detection  and  recognition.  However,  wide-spread  use  of  polarization  data  in  the  infrared 
has  not  occurred,  probably  because  polarization-sensitive  imaging  methods  have  been  slow, 
bulky,  and  subject  to  object  misregistration.  Physics  Innovations  Inc.  is  developing  polarization- 
sensitive  imaging  sensors  which  are  compact,  portable,  and,  for  some  applications,  can  be 
retrofitted  into  existing  optical  imaging  systems. 

In  this  section,  we  describe  the  operation  of  polarization-sensitive  thermal  imaging 
sensors  which  are  capable  of  simultaneously  capturing  intensity  and  polarization  data  without 
misregistration  and  in  real  time.  The  results  show  that  three-dimensional  information,  in 
addition  to  temperature  distribution,  can  be  captured  and  displayed  in  real  time  and  at  video 
frequency.  These  results  also  show  that  polarization  data  provides  useful  contrast,  enough  to 
distinguish  man-made  objects  from  backgrounds  and  to  distinguish  different  surface  orientations. 

In  Section  2,  we  describe  a  modification  to  our  method  of  polarization-sensitive  thermal 
imaging.  With  this  modification,  images  of  circular  polarization  can  be  captured.  Polarimetric 
imaging  is  often  proposed  as  a  technique  to  improve  target  detection  and  recognition. 

In  Section  3  we  describe  our  investigation  into  the  improvement  in  performance  of 
automatic  target  recognition  algorithms  when  processing  polarimetric  images  compared  to  FLIR 
images. 

Polarized  thermal  emission  can  be  explained  by  relating  to  the  more  common  experience 
of  polarized  reflection.  When  light  is  incident  at  a  grazing  angle  on  the  surface  of  water  the 
portion  of  the  light  which  is  reflected,  as  in  sun  glint,  is  linearly  polarized  with  a  horizontal  plane 
of  polarization.  (Polarizers  in  sunglasses  are  oriented  to  attenuate  horizontally  polarized  light.) 
The  portion  of  the  light  which  is  transmitted  into  the  water  is  linearly  polarized  with  a  vertical 
plane  of  polarization.  By  the  same  mechanism,  if  light  originates  below  the  surface  and  passes 
through  the  surface,  the  transmitted  light  would  be  linearly  polarized  with  a  vertical  polarization 
plane.  The  polarization  plane  contains  the  normal  vector  to  the  surface.  The  transmitted  light 
when  exiting  at  a  more  grazing  angle  to  the  surface  will  have  a  greater  percent  of  polarization. 

Thermal  emission  is  light  originating  below  the  surface.  Referring  to  Figure  1,  when 
thermal  radiation  is  received  by  a  polarization-sensitive  sensor  from  a  surface  element  on  an 
object,  the  plane  of  polarization  gives  the  angle  (|)  and  the  percent  of  polarization  P  can  be  related 
to  the  angle  0.  For  most  materials,  P  increases  as  0  increases  [Figure  2].  Together  0  and  (|) 
determine  the  normal  vector  to  the  surface  element. 

1.2.  IMAGING  OF  STOKES  PARAMETERS 

A  beam  of  incoherent  radiation  emitted  or  reflected  from  a  target's  surface  can  be 
completely  described  at  a  given  wavelength  by  the  four  Stokes  parameters,  (I,  Q,  U,  V).  The 
first  Stokes  parameter  I  is  a  measure  of  the  total  intensity  of  radiation.  The  second  parameter  Q 
measures  the  amount  of  linear  polarization  in  the  horizontal  direction.  The  third  parameter  U 
measures  the  amount  of  linear  polarization  in  a  plane  rotated  45  degrees  from  the  horizontal. 


The  fourth  parameter  V  is  associated  with  the  circular  polarization.  A  method  for  imaging  V  is 
described  in  Section  2. 

The  first  three  Stokes  parameters  can  be  transformed  into  percent  of  polarization  P  and 
angle  of  polarization  (|)  using  the  relations,1 
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Conventional  methods  of  determining  polarization  from  images  rely  on  the  use  of  a  single 
polarizer  covering  the  entire  imaging  sensor.  In  one  method,  a  sequence  of  four  images  is  taken 
with  a  linear  polarizer  oriented  at  0°,  45°,  90°,  and  135°.  This  method  can  determine  the  first 
three  of  the  Stokes  parameters  at  each  image  pixel: 

I  =  “ *(i<)  +i45  +i90  i  135 ) 

Q  =  i0  -i9o  (2) 

U  =  145  i  135 

where  ix  is  the  intensity  measured  with  the  polarizer  oriented  at  x  degrees.  These  equations 
assume  that  the  polarizers  are  ideal,  i.e.  100%  transmittance  for  radiation  linearly  polarized  at 
angle  x  and  zero  transmittance  for  radiation  at  angle  x  +  90°.  The  effects  of  non-ideal  polarizers 
are  discussed  in  Reference  1 . 

1.3.  METHOD  OF  IMAGING  INTENSITY  AND  POLARIZATION  IN  A  SINGLE 

VIDEO  FRAME 

In  an  imaging  sensor  developed  at  Physics  Innovations  Inc.,  each  detector  element  has  its 
own  polarizer  and  the  polarizer  orientations  are  rotated  from  element  to  element.  Referring  to 
Figure  3,  the  Stokes  parameters  characterizing  radiation  emitted  and  reflected  from  the  part  of  an 
object  imaged  onto  detector  elements  A,  B,  C,  and  D  can  be  determined  by  using  the  Equations 
(2).  The  accuracy  of  this  measurement  assumes  that  the  intensity  and  polarization  state  does  not 
vary  significantly  over  A,  B,  C,  and  D.  This  assumption  is  satisfied  if  detector  elements  A,  B,  C, 
and  D  see  only  a  localized  part  of  an  object.  Localized  measurements  can  be  made  if  the 
detector  array  has  high  spatial  resolution  relative  to  the  overall  dimensions  of  the  object.  (This 
assumption  is  not  necessary  when  using  the  technique  described  in  the  next  section.) 

Wire  grid  polarizers  can  be  integrated  with  photodiode  arrays  by  placing  the  polarizers 
on  a  substrate  in  close  proximity  to  the  photodiode  array  [Figure  4(a)].  The  optimum 
arrangement  would  have  the  polarizer  array  formed  on  the  same  substrate  as  the  photodetector 
focal  plane  array,  i.e.  a  monolithic  integration.  For  the  purpose  of  demonstrating  the  feasibility 
of  using  polarizer  arrays,  a  more  convenient  method  of  integrating  the  polarizer  array  with  a 
thermal  imaging  sensor  is  shown  schematically  in  Figure  4(b).  Light  from  the  target  is  imaged 
by  an  objective  lens  onto  the  polarizer  array.  The  light,  which  is  transmitted  through  the 
polarizers,  is  imaged  using  a  relay  lens  onto  the  thermal  imaging  sensor,  here  a  focal  plane  array. 
In  the  work  described  in  this  paper,  the  focal  plane  array  camera  was  a  Mitsubishi  IR-M300 


Thermal  Imager.  The  Mitsubishi  camera  has  a  256x256  PtSi  array  and  is  sensitive  to  radiation  in 
the  3  to  5  micron  region  (MWIR).  The  technique  shown  in  Figure  4(b)  can  also  be  used  to 
integrate  polarizer  arrays  with  other  types  of  imaging  sensors,  such  as  LWIR  cameras  and 
uncooled  focal  plane  arrays. 

Figure  5  is  an  image  of  a  heated  cube  which  is  used  as  a  target.  A  single  video  frame  was 
captured  from  the  output  of  the  Mitsubishi  camera.  A  computer  program  was  written  which 
takes  the  video  frame  and  calculates  the  Stokes  parameters  I,  P,  and  <|>  using  Equations  (1)  and 
(2). 

1.4.  METHOD  OF  IMAGING  INTENSITY  AND  POLARIZATION  IN  A  SEQUENCE 

OF  VIDEO  FRAMES 

The  above  images  of  intensity  and  polarization  do  not  have  the  full  spatial  resolution  of 
the  focal  plane  camera,  256  x  256  for  the  Mitsubishi  M300  camera.  The  full  spatial  resolution 
can  be  achieved  by  microscanning  the  image  over  the  polarizer  array.  As  shown  in  Figure  7, 
first  a  video  frame  is  captured,  then  the  image  is  displaced  horizontally  and  a  second  video  frame 
is  captured.  The  image  is  displaced  vertically  and  a  third  video  frame  is  captured.  From  these 
three  video  frames,  the  Stokes  parameters  can  be  calculated.  For  radiation  from  a  point  source 
imaged  onto  the  circle  in  Figure  7(a)  the  Stokes  parameters  are 

I  =  i45  +  i]35 

Q  =  (— 2  •  4o)  +  I45  +  il35 

U  =  i45  -il35 

Using  Equations  (1),  images  of  I,  P,  and  (])  can  be  calculated. 

To  microscan  the  image,  we  used  two  tilted  plane-parallel  plates  each  driven  by  a 
stepping  motor.  One  plate  and  motor  were  used  for  horizontal  displacement,  and  the  other  plate 
and  motor  for  vertical  displacement.  The  motors  were  combined  with  a  microstepper  driver  for 
sub-pixel  resolution  and  an  indexer/controller  which  interfaces  with  a  personal  computer.  The 
motor  can  rotate  a  plate  by  a  few  degrees  in  less  than  1/60  sec  which  is  a  frame  time  for  the 
Mitsubishi  thermal  imager.  With  high-speed  digital  signal  processors,  we  should  be  able  to 
capture  the  sequence  of  three  frames  in  about  6T/60  sec.  However,  in  the  prototype  sensor,  such 
processors  were  not  available,  and  the  three  frames  were  captured  in  about  3  sec. 

Using  the  microscanning  method,  we  captured  polarization  data  on  a  die-cast  metal  scale 
model  of  an  aircraft  [Figure  6].  The  model  is  painted  a  flat  black  and  heated  to  about  40C.  The 
intensity  I  image  shows  the  temperature  distribution  over  the  target  and  is  identical  to  an  image 
from  a  FLIR  sensor.  The  P  image  shows  significant  polarization  (bright  pixels)  over  the  surfaces 
which  are  viewed  at  a  grazing  angle.  Surfaces  facing  the  sensor,  around  the  nose  of  the  aircraft, 
have  lower  percent  of  polarization  (darker  pixels).  Darker  pixels  are  also  seen  below  the  V- 
shaped  tail.  This  is  probably  due  to  radiation  emitted  by  the  tail  and  reflected  off  the  body  of  the 
aircraft.  When  reflected  and  emitted  radiation  are  combined,  the  percent  of  polarization  is 
reduced  from  that  of  emission  alone. 

In  the  image  of  the  angle  of  polarization  <\>,  surfaces  which  face  the  observer’s  right  are 
dark  gray  and  surfaces  facing  the  observer’s  left  are  light  gray  [Figure  6].  The  vertical  stripe 
pattern  in  this  (|>  image  is  caused  by  nonuniformities  in  the  particular  polarizer  array  used. 


2.  CIRCULAR  POLARIZATION  IMAGING  USING  AN  ACHROMATIC  RETARDER 

ARRAY 


2.1.  ACHROMATIC  QUARTER-WAVE  RETARDER 

The  sensor  described  in  the  previous  section  is  capable  of  imaging  three  Stokes 
parameters.  In  order  to  extend  the  sensor’s  capabilities  to  image  all  four  Stokes  parameters  I,  Q, 
U,  and  V,  we  are  developing  novel  microscale  retarder  arrays.  In  an  imaging  sensor,  each 
retarder  element  will  be  combined  with  a  linear  polarizer  element.  This  sensor  will  be  able  to 
capture  images  of  the  four  Stokes  parameters  in  real  time  and  without  mechanically  moving 
parts.  The  waveplates  and  polarizers  can  be  fabricated  using  conventional  microlithography. 

A  well-established  method  for  constructing  an  achromatic  quarter-wave  retarder  is  to  use 
a  combination  of  a  quarter- wave  waveplate  followed  by  a  half-wave  waveplate3.  In  Figure  8(a), 
a  Poincare  sphere  representation  shows  how  the  composite  retarder  transforms  right  circularly 
polarized  light  V  to  horizontal  linearly  polarized  light  Q.  The  quarter-wave  waveplate 
transforms  the  circularly  polarized  component  V  to  linearly  polarized  component  A.  The  half¬ 
wave  waveplate  then  rotates  the  plane  of  polarization  to  change  component  A  to  Q. 

The  composite  retarder  is  achromatic.  For  example,  if,  at  a  different  wavelength,  the  magnitude 
of  the  birefringence  is  slightly  larger  by  a  factor  m  >1,  then,  on  the  Poincare  sphere,  the 
component  V  will  be  transformed  through  a  path  m  times  longer  to  A'.  In  the  half-wave 
waveplate,  the  path  will  also  be  m  times  longer,  so  A'  will  be  transformed  to  a  point  close  to  Q. 

The  composite  retarder  may  also  have  a  wider  field  angle  than  a  single  waveplate.  If  a 
ray  enters  the  quarter-wave  waveplate  in  a  direction  normal  to  the  surface,  then  V  will  be 
transformed  to  A.  If  another  ray  is  incident  at  an  angle  greater  than  zero,  then  it  travels  a  longer 
distance  through  the  waveplate,  and  V  will  be  transformed  to  A'.  The  latter  ray  will  also  travel 
through  a  longer  distance  in  the  half-wave  waveplate,  and  A'  will  be  transformed  to  a  point  close 
to  Q. 

The  composite  retarder  can  be  constructed  by  stacking  two  waveplates  with  their  fast 
axes  appropriately  rotated  [Figure  8(b)].  As  discussed  below,  an  array  of  composite  retarders 
can  be  fabricated  on  a  substrate. 

2.2.  SURFACE-RELIEF  GRATINGS 

In  the  sensor  described  in  Section  1,  each  pixel  in  a  focal  plane  array  has  its  own 
microscale  linear  polarizer.  Correspondingly,  in  order  to  detect  circular  polarized  light,  the 
linear  polarizer  in  Figure  8(b)  has  its  own  microscale  composite  retarder.  The  composite 
retarder  is  an  element  in  an  array  of  retarders.  Each  composite  retarder  can  be  constructed  by 
stacking  two  waveplates,  The  individual  waveplates  are  surface-relief  gratings.  A  grating  is  a 
series  of  identical  parallel  linear  ridges  arranged  on  a  planar  substrate.  As  shown  in  Figure  9, 
these  grooves  and  ridges  constitute  a  form  birefringent  material. 

We  demonstrated  that  a  combination  of  surface-relief  gratings  behaves  as  an  achromatic 
quarter-wave  retarder  using  Mueller  matrices.4  We  assume  a  Si/air  layered  material  for  the  14- 
wave  and  14-wave  waveplates.  We  can  calculate  the  birefringence  from  effective  medium 
theory.5  For  radiation  incident  in  a  direction  normal  to  plane  of  the  surface-relief  gratings,  the 
retardance,  for  radiation  with  wavelengths  from  3  to  5  pm,  is  shown  in  Figure  1 1(a).  The 
retardance  is  nearly  independent  of  wavelength  in  the  range  3-5  pm  with  a  standard  deviation  of 
0.626°.  For  comparison,  the  retardance  of  a  14-wave  waveplate  made  from  a  single  surface  relief 
grating  layer  is  plotted  and  has  a  standard  deviation  of  12.5°. 


In  our  imaging  sensor,  light  will  enter  the  composite  retarder  in  a  range  of  angles  [Figure 
10(c)].  The  field  angle  required  for  the  composite  retarder  is  determined  by  the  f-number  of  the 
objective  lens.  For  example,  if  f/#  =  1.2,  then  maximum  field  angle  is  22.6°.  The  Mueller 
matrix  calculation,  assuming  an  incidence  angle  of  22.6°  from  normal,  is  shown  in  Figure  1 1(b). 
The  retardance  has  a  standard  deviation  of  0.712°  for  the  composite  retarder.  We  varied  the 
angles  of  the  fast  axes  [Figure  8(b)],  and  found  that  the  optimum  angles  are  -30.0°  for  the  14- 
wave  grating  and  +29.3°  for  the  !4-wave  grating.  These  are  the  angles  used  in  Figure  11. 

2.3.  IMAGING  FOUR  STOKES  PARAMETERS 

Figure  10  shows  how  a  wide-band  Stokes-vector  imaging  sensor  can  be  constructed  by 
placing  the  retarder  array  in  close  proximity  with  the  wire  grid  polarizer  array  in  a  sandwich 
structure.  Light  from  the  target  is  focused  by  an  objective  lens  on  the  retarder/polarizer  arrays. 
Light  which  is  transmitted  by  the  retarder/polarizer  arrays  is  imaged  onto  the  focal  plane  array 
using  a  relay  lens.  In  Figure  10(a),  when  array  elements  A,  B,  C,  D,  E,  and  F  view  a  localized 
area  on  the  target,  then  the  Stokes  images  can  be  captured  in  a  single  video  frame.  The 
photodetector  signals  at  A,  B,  C,  D,  E,  and  F  give  the  intensities  i(90°,0°),  i(135°,0°), 
i(45o,0°),i(0o,0°),  i(90°,90°),  and  i(0°,90°),  respectively,  where  the  first  argument  is  the  angle  of 
rotation  of  the  wire  grid  polarizer  and  the  second  is  the  phase  change  due  to  the  retarder.  For 
ideal  polarizers  and  retarders,  the  Stokes  parameters  are 

I  =  i(0,0)  +  i(90,0) 

Q  =  i(0,0)  -  i(90,0) 

U  =  i(45,0)  -  i(l  35,0) 

V  =  i(0,90)  -  i(90,90) 


3.  ATR  ALGORITHM  PERFORMANCE  IMPROVEMENT  USING  POLARIMETRIC 

IMAGING 

3.1.  SYNTHETIC  IMAGERY 

Images  with  polarization  data  are  presently  available  with  the  sensor  developed  at 
Physics  Innovations  Inc.  [Figure  5  and  Figure  6].  However,  at  the  time  of  this  Automatic  Target 
Recognition  (ATR)  study,  actual  polarization  data  did  not  exist,  so  we  generated  synthetic 
images  using  ray  tracing.6  A  publicly  available  ray-tracing  package  was  modified  with 
additional  modules  to  generate  polarization  imagery.  In  this  study  the  target  is  a  tactical  missile 
mounted  horizontally  on  top  of  a  mobile  launcher  [Figure  12].  The  dimensions  of  this  missile 
and  launcher  are  approximated  those  of  the  Russian-built  SCUD  missile. 

The  sensor-derived  image  was  generated  using  the  ray  tracer  program  with  the  sensor 
noise  included.  An  example  of  a  sensor  image  is  shown  in  Figure  12.  In  all  the  sensor  images 
the  background  is  a  planar  surface  with  the  same  emissivity  and  the  same  temperature  (300  K)  as 
the  surface  of  the  missile  and  mobile  launcher.  The  sensor  characteristics  are  typical  of  a  PtSi 
focal  plane  array  camera.  For  the  images  discussed  in  this  paper  the  frame  integration  time  is  33 
millisecond,  similar  to  video  frequencies. 


3.2.  EDGE  DETECTING  ALGORITHM 

A  selection  of  algorithms  for  target  edge  detection,  segmentation,  and  recognition  were 
evaluated  using  the  sensor  images  and  model  images  generated  with  the  polarization  ray-tracing 
program.  The  polarization  images  have  three  components:  intensity  I,  percent  of  polarization  P, 
and  angle  of  the  plane  of  polarization  (|).  The  intensity  I  image  is  the  same  as  would  be  captured 
using  a  FLIR  sensor.  We  demonstrated  the  advantages  of  using  polarization  imagery  by 
comparing  the  performance  of  the  algorithms  on  intensity-only  (FLIR)  data  and  the  performance 
on  a  combination  of  intensity  and  polarization  data. 

For  edge  detection,  we  used  a  Sobel  operator.  The  evaluation  method  can  be  described 
as  follows.  (1)  apply  the  Sobel  operator  on  each  channel  (I,  P,  and  <\>)  of  the  sensor-derived 
images  (2)  apply  thresholds  to  each  channel  of  the  Sobel  images,  (3)  add  (logical  OR)  the 
images  from  each  channel  to  form  a  single  binary  image,  and  (4)  make  a  pixel-by-pixel 
comparison  of  this  binary  image  with  the  image  of  the  true  edges.  For  the  case  where  we 
evaluate  performance  on  intensity-only  (FLIR)  data,  there  is  only  one  channel  (I).  The  detection 
rate  is  defined  as  the  ratio  of  the  number  of  true  edge  pixels  detected  to  the  number  of  actual  true 
edge  pixels.  The  false  detection  rate  is  defined  as  the  ratio  of  the  number  of  pixels  labeled  as 
edges  which  are  not  true  edges  to  the  number  of  pixels  which  are  not  true  edges. 

The  performance  of  the  Sobel  operator  on  the  sensor  images  is  summarized  using  receiver- 
operating-characteristics  (ROC)  curves  [Figure  13]  where  the  edge  detection  rate  is  plotted 
versus  the  false  detection  rate.  Different  points  along  a  ROC  curve  correspond  to  different 
choices  of  thresholds  when  labeling  pixels  as  edges. 

The  performance  evaluation  showed  that  polarization  data  contribute  an  improvement 
over  most  aspect  and  elevation  angles.  For  example,  at  30°  elevation  angle  and  aspect  angle  of 
30°,  polarization  data  gives  at  least  16%  improvement  in  detection  probability  over  intensity- 
only  (FLIR)  data  alone  [Figure  13].  However,  this  improvement  decreases  to  4%  when  the 
target  is  viewed  from  directly  overhead.  When  the  target  is  viewed  directly  broadside  or  for 
front  or  back  views,  polarization  data  does  not  improve  edge  detection  over  FLIR  alone. 

3.3.  STATISTICAL  SEGMENTATION  AND  LABELING  ALGORITHM 

In  the  statistical  segmentation  and  labeling  algorithm,  the  input  consists  of  the  sensor  I,  P, 
and  (|)  imagery,  the  estimated  size  of  the  targets,  and  the  first  and  the  second  order  statistics  of  the 
objects  of  the  interest.  The  first  and  the  second  order  statistics  of  the  regions  of  the  original 
imagery  falling  under  a  moving  window  of  the  size  specified  by  the  estimated  size  of  the  objects 
of  interest  are  computed.  Next  a  measure  of  the  statistical  similarity,  the  Fisher  Criterion  F, 
between  the  extracted  statistics  and  the  inputted  statistics  are  made.  The  regions  that  have 
statistical  similarity  to  the  object’s  statistics  will  generate  low  F  values.  This  algorithm  will 
produce  not  only  segmentation  of  the  scene  but  also  a  classification  of  them  into  classes  of 
interest. 

The  method  used  to  evaluate  the  performance  of  this  algorithm  has  the  following  steps. 
(1)  Apply  the  statistical  segmentation  and  labeling  algorithm  to  the  sensor  image.  The  output  is 
a  grayscale  image,  F  image,  where  each  pixel  is  labeled  with  a  value  F,  a  measure  of  the 
statistical  similarity  between  the  extracted  statistics  of  the  sensor  image  and  the  statistics  of  the 
true  image  of  the  target.  (2)  Threshold  the  F  image;  all  pixels  with  F  less  than  the  threshold  are 
labeled  to  be  target  pixels.  (3)  Compare  pixel-by-pixel  the  thresholded  F  image  with  the  model 
image  containing  the  true  target  region.  The  results  of  the  comparison  are  described  in  terms  of 


segmentation  accuracy  and  false  detection  rate.  Segmentation  accuracy  is  defined  as  the  ratio  of 
the  number  of  pixels  in  the  set  (L  n  T)  to  the  number  of  pixels  in  the  set  (L  u  T),  where  L  is 
the  set  of  pixels  labeled  to  be  target  pixels  and  T  is  the  set  of  pixels  in  the  true  target  region. 
False  detection  rate  is  defined  as  the  ratio  of  the  number  of  pixels  labeled  to  be  target  pixels 
which  are  not  in  the  true  target  region  to  the  number  of  pixels  not  in  the  true  target  region  (true 
background  pixels). 

The  performance  evaluation  shows  that,  over  large  ranges  of  elevation  and  aspect  angles, 
the  use  of  polarization  data  improves  segmentation  accuracy  significantly  over  the  use  of 
intensity  (FLIR)  data  alone.  The  performance  of  the  statistical  segmentation  and  labeling 
algorithm  was  plotted  as  segmentation  accuracy  versus  false  detection  rate.  The  points  along 
curves  correspond  to  different  threshold  values.  As  shown  in  Figure  14,  there  is  a  dramatic 
improvement  in  segmentation  accuracy,  as  much  as  55%,  when  using  polarization  data  near  the 
elevation  angle  of  30°  and  aspect  angle  of  30°.  For  the  overhead  view,  the  improvement  is  as 
large  as  20%.  Flowever,  for  front,  back,  and  broadside  views,  there  is  no  improvement  when 
using  polarization  data. 

3.4.  TARGET  CLASSIFICATION  ALGORITHM 

3.4.1.  CLASSIFICATION  TECHNIQUES 

There  are  a  number  of  ways  that  objects  can  be  classified.  Among  these  approaches,  are 
feature-based  methods  and  model-based  techniques.  In  the  feature-based  methods,  a  relatively 
small  set  of  features,  capturing  the  unique  characteristics  of  the  targets,  are  extracted  from  the 
potential  targets.  Then  they  are  compared  with  the  stored  sets  of  similar  feature  types  that  are 
associated  with  known  targets  of  interest.  A  decision  is  made  based  on  the  statistical  similarity 
of  the  two.  In  the  model-based  techniques,  the  classification  is  viewed  as  the  output  of  two 
processes  of  hypothesis  generation  and  hypothesis  verification.  Based  on  the  sensory-derived 
object  signature  a  set  of  hypothesis  about  the  object  labels  will  be  generated.  Then,  for  each 
hypothesis,  use  will  be  made  of  everything  that  is  known  about  the  target  given  that  the 
hypothesis  is  true.  This  knowledge  includes  a  physics-based  model  of  the  target  under  the 
existing  scene  and  scenario.  Then,  this  expected  knowledge  is  searched  and  verified  in  the 
actual  sensory-derived  signature  and  a  similarity  measure  is  computed.  This  process  will  be 
performed  for  each  hypothesis.  This  reasoning  mechanism  will  continue  until  one  achieves 
acceptable  level  of  confidence  in  one  or  more  hypothesis.  The  preference  for  the  approach  is 
dependent  on  a  number  of  parameters,  among  them  the  availability  of  the  models,  number  of 
pixels  on  targets,  and  computational  requirements. 

The  approach  that  is  pursued  here  is  a  model-based  technique.  The  output  of  the 
Detection  Module  is  fed  to  the  Recognition  Module  that  matches  the  edge  map  and  the 
segmented  regions  with  the  stored  polarization-sensitive  models  of  the  objects  of  interest.  The 
recognition  is  performed  in  two  ways.  In  one  approach  that  was  described  previously,  a 
statistical  similarity  measure  (Fisher  Criterion)  is  used  to  match  the  extracted  target's  with  a 
library  of  stored  prototypes.  In  another  approach,  an  image  correlator  is  used  to  correlate  the 
stored  models  with  the  edge  map  and  the  segmented  target  regions.  A  correlation  surface  is 
obtained  and  then  the  correlation  distances  between  the  best  matches  corresponding  to  different 
object  models  are  compared  to  decide  on  the  best  label  for  the  unknown  objects.  Figure  15(a) 
shows  a  target  model,  the  original  sensory-derived  images  of  Figure  12  with  the  polarization 
components  represented  as  color.  Figure  15(b)  shows  a  portion  of  the  tail  of  the  missile.  Figure 


15(c)  shows  the  resulting  correlation  surface  with  the  position  of  the  correct  match.  The  arrow  at 
the  right  inside  the  Figure  15(c)  indicates  the  position  of  the  highest  peak. 

3.4.2.  PERFORMANCE  EVALUATION  OF  CLASSIFICATION  ALGORITHM 

In  the  model-based  recognition  scheme,  the  sensory  derived  object  was  matched  against  a 
set  of  stored  target  models,  and  the  maximum  matching  distance  was  tabulated.  The  sensory 
derived  objects  corresponded  to  views  of  the  missile  on  a  mobile  launcher  [Figure  12]  seen  from 
an  elevation  angle  of  30  degrees  and  aspect  angles  of  0,  30,  90,  130,  and  180  degrees.  The 
resulting  confusion  matrix  is  shown  in  Tables  1-3. 

Tables  1-3  show  that  the  intensity  I-only  sensor  data,  i.e.  FLIR  images,  cannot  correctly 
classify  among  most  of  the  target  aspect  angles.  However,  with  a  combination  of  percent  of 
polarization  P  and  angle  of  polarization  (|>,  all  aspect  angles  were  correctly  classified  (with  the 
exception  of  90  degrees:  broadside  view).  This  demonstrates  that  a  combination  of  intensity  and 
polarization  data  can  provide  a  higher  performance  than  the  intensity  data  alone  can  provide. 

The  bold  numbers  show  the  maximum  value  for  each  row. 


Table  1.  Confusion  Matrix:  Matching  results  for  intensity  I-only  (FLIR). 


Model  at  130° 
Aspect  Angle 

Model  at  180° 
Aspect  Angle 

Sensory-derived 
Target  at  0° 
Aspect  Angle 

0.915 

0.754 

0.817 

0.799 

0.954 

Sensory-derived 
Target  at  30° 
Aspect  Angle 

0.913 

0.741 

0.815 

0.792 

0.954 

Sensory-derived 
Target  at  90° 
Aspect  Angle 

0.919 

0.750 

0.841 

0.797 

0.954 

Sensory-derived 
Target  at  130° 
Aspect  Angle 

0.916 

0.742 

0.818 

0.807 

0.953 

Sensory-derived 
Target  at  180° 
Aspect  Angle 

0.917 

0.742 

0.815 

0.790 

0.964 

Table  2.  Confusion  Matrix:  Matching  results  for  angle  of  polarization  plane  c|)-only. 


Model  at  0° 
Aspect  Angle 

Model  at  30° 
Aspect  Angle 

Model  at  90° 
Aspect  Angle 

Model  at  130° 
Aspect  Angle 

Model  at  180° 
Aspect  Angle 

Sensory-derived 
Target  at  0° 
Aspect  Angle 

0.667 

0.488 

0.204 

0.546 

0.511 

Sensory-derived 
Target  at  30° 
Aspect  Angle 

0.570 

0.642 

0.215 

0.522 

0.497 

Sensory-derived 
Target  at  90° 
Aspect  Angle 

0.599 

0.499 

0.216 

0.550 

0.460 

Sensory-derived 
Target  at  130° 
Aspect  Angle 

0.568 

0.510 

0.232 

0.675 

0.477 

Sensory-derived 
Target  at  180° 
Aspect  Angle 

0.585 

0.499 

0.207 

0.547 

0.524 

Table  3.  Confusion  Matrix:  Matching  results  for  percent  of  polarization  P-only. 


Model  at  0° 
Aspect  Angle 

Model  at  30° 
Aspect  Angle 

Model  at  90° 
Aspect  Angle 

Model  at  130° 
Aspect  Angle 

Model  at  180° 
Aspect  Angle 

Sensory-derived 
Target  at  0° 
Aspect  Angle 

0.836 

0.627 

0.542 

0.630 

0.785 

Sensory-derived 
Target  at  30° 
Aspect  Angle 

0.788 

0.653 

0.542 

0.629 

0.776 

Sensory-derived 
Target  at  90° 
Aspect  Angle 

0.782 

0.623 

0.583 

0.616 

0.774 

Sensory-derived 
Target  at  130° 
Aspect  Angle 

0.786 

0.619 

0.544 

0.638 

0.774 

Sensory-derived 
Target  at  180° 
Aspect  Angle 

0.794 

0.620 

0.541 

0.627 

0.823 

4.  SUMMARY 

Two  methods  have  been  described,  where  intensity  and  polarization  images  are  captured 
and  processed  into  images  of  the  target's  temperature  distribution  and  the  target's  three- 
dimensional  shape  and  orientation.  In  the  first  method,  a  single  video  frame  is  captured  and 
processed  into  images  of  temperature  and  shape.  In  the  second  method,  microscanning  is  used 
and  three  video  frames  are  captured  and  processed  into  images  of  temperature  and  shape  with 
higher  resolution.  The  first  method  would  require  almost  no  changes  to  the  optics  and 


mechanical  structure  of  existing  sensors  employing  focal  plane  arrays.  When  shape  and 
temperature  are  varying,  the  polarization-sensitive  thermal  imaging  sensor  will  be  able  to  capture 
these  changes  in  real  time.  With  high-speed  signal  processing,  high-resolution  polarimetric  data 
can  be  captured  and  displayed  at  video  frequency. 

In  order  to  extend  the  sensor’s  capabilities  to  image  all  four  Stokes  parameters  I,  Q,  U, 
and  V,  we  are  developing  novel  microscale  retarder  arrays.  These  retarders  will  be  combined 
with  linear  polarizers  in  an  imaging  sensor.  This  sensor  will  be  able  to  capture  images  of  the 
four  Stokes  parameters  in  real  time  and  without  mechanically  moving  parts.  Each  retarder  is  a 
composite  of  surface-relief  gratings.  We  have  demonstrated,  by  calculation,  that,  with  the  proper 
dimensions  and  orientation,  the  composite  retarder  is  highly  achromatic  and  has  a  large  field 
angle. 

Polarimetric  imaging  is  often  proposed  as  a  technique  to  improve  target  detection  and 
recognition.  We  have  demonstrated,  using  a  limited  set  of  synthetic  imagery,  a  significant 
improvement  in  the  performance  of  automatic  target  detection,  segmentation,  and  classification 
algorithms  when  processing  polarimetric  images  compared  to  FLIR  images. 
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Polarization-sensitive 
thermal  sensor 


Figure  1.  Thermal  radiation  from  the  surface  of  an  object  contains  information  about  the  object’s 
shape.  By  sensing  polarization  data  from  a  point  on  the  surface,  the  orientation  of  the  surface 
element  can  be  derived.  The  normal  vector  to  the  surface  element  is  n  which  is  defined  by  the 
angles  0  and  (f)  is  imaged  directly.  0  is  calculated  from  the  image  of  the  percent  of 
polarization  P. 


Figure  2.  For  most  materials,  the  percent  of  polarization  P  increases  as  the  radiation  is  emitted  at 
a  more  grazing  angle,  that  is  at  greater  0. 


Figure  3.  Physics  Innovations  Inc.  has  developed  a  polarization-sensitive  thermal  imaging 
sensor.  In  this  sensor  each  detector  element  of  a  focal  plane  array  has  its  own  polarizer  and  the 
polarizer  orientations  are  rotated  from  element  to  element.  The  polarizers  are  wire  grids  formed 
by  microlithography  on  a  substrate. 
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Figure  4.  Two  methods  for  integrating  the  polarizer  array  and  focal  plane  array,  (a)  Wire  grid 
polarizers  can  be  integrated  with  photodiode  arrays  by  placing  the  polarizers  on  a  substrate  in 
close  proximity  to  the  photodiode  array,  (b)  A  schematic  diagram  of  the  polarization-sensitive, 
thermal  imaging  sensor.  Radiation  from  the  target  is  imaged  by  an  objective  lens  onto  the 
polarizer  array.  Light  which  is  transmitted  by  the  polarizer  array  is  imaged  by  a  relay  lens  onto 
the  photodetector  array. 
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Figure  5.  The  object  is  a  heated  cube.  Images  of  three  Stokes  parameters,  calculated  from  a 
single  video  frame.  These  I,  P,  and  <\>  images  demonstrate  that  the  three-dimensional  shape  of  a 
target  and  the  target's  temperature  distribution  can  be  obtained  simultaneously  using 
polarization-sensitive  thermal  imaging. 
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Figure  6.  High-resolution  polarization  image  of  a  scale  model  of  an  aircraft.  High-resolution 
was  achieved  by  using  a  microscanning  technique  described  in  Figure  7. 
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Figure  7.  The  sensor  shown  in  Figure  4(b)  was  improved  with  higher  resolution  and  the  ability 
to  resolve  polarimetric  data  from  a  point  source,  e.g.  targets  at  long  range.  This  improvement 
uses  the  technique  of  microscanning,  (a)  In  three  successive  video  frames,  the  image  on  the 
polarizer  array  is  displaced  horizontally  and  vertically.  The  circle  represents  the  image  of  a 
point  source.  By  combining  Frames  1,  2  and  3,  the  three  parameters  I,  P  and  can  be 
determined  for  the  point  source.  I,  P,  and  <\>  can  be  captured  and  displayed  at  video  frequency, 
(b)  The  image  can  be  microscanned  by  tilting  a  plane  parallel  plate  in  the  optical  path  about 
vertical  and  horizontal  axes. 


(a) 

Figure  8.  A  well-established  method  for  constructing  an  achromatic  quarter- wave  retarder  is  to 
use  a  combination  of  a  quarter-wave  waveplate  followed  a  half-wave  waveplate.  (a)  This 
Poincare  sphere  representation  shows  how  the  composite  retarder  transforms  circularly  polarized 
light  V  to  linearly  polarized  light  Q.  (b)  In  a  pixel  of  the  4-Stokes-parameter  imaging  sensor,  a 
composite  retarder  will  be  combined  with  a  linear  polarizer. 


Figure  9.  .  A  surface-relief  grating  is  a  series  of  identical  parallel  linear  ridges  arranged  on  a 
planar  substrate.  The  grating  can  be  seen  to  have  form  birefringence  by  considering  the  grating 
as  a  thin  slice  of  a  periodic  stratified  material.  These  gratings  can  be  used  to  form  a  waveplate 
array  in  the  4-Stokes-parameter  imaging  sensor  [Figure  10]. 
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Figure  10.  A  wide-band  sensor  for  imaging  four  Stokes  parameters,  (a)  A  retarder  array  can  be 
combined  with  a  polarizer  array,  (b)  The  retarder  array  will  contain  elements  which  are 
achromatic  quarter- wave  retarders.  Stokes  vector  images  can  be  captured  using  the  arrangement 
shown  in  (c).  The  retarder  array  and  polarizer  array  are  placed  at  the  focus  of  an  objective  lens. 
Light  which  is  transmitted  by  the  arrays  is  focused  onto  a  focal  plane  array  using  a  relay  lens. 


Figure  1 1 .  (a)  Retardance  for  a  single  !4-wave  waveplate  (  )  and  for  the  composite  retarder  (x) 
at  normal  incidence,  (b)  Retardance  at  incidence  angle  22.6°  from  normal.  These  results  show 
that,  in  the  wavelength  range  3-5  pm,  the  retardance  of  the  composite  retarder  is  nearly 
independent  of  wavelength  and  varies  negligibly  over  field  angles  subtended  by  a  low  f/#  lens. 
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Figure  12.  For  our  study  of  automatic  target  detection  algorithms,  a  polarization  ray  tracer  was 
used  to  generate  the  sensor  images.  The  sensor  images  include  the  characteristics  of  a  typical 
PtSi  focal  plane  array  camera.  The  image  has  three  components,  intensity  I  (same  as  FLIR), 
percent  of  polarization  P,  and  angle  of  polarization  (|). 


Figure  13.  The  performance  of  the  Sobel  operator  on  the  sensor  images  is  summarized  using 
receiver-operating-characteristics  (ROC)  curves 
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Figure  14.  The  performance  of  the  statistical  segmentation  and  labeling  algorithm  is  measured 
by  the  segmentation  accuracy  and  false  detection  rate. 
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Figure  15.  (a)  The  original  sensory-derived  images  of  Figure  12  with  the  polarization 
components  represented  as  color,  (b)  A  portion  of  the  tail  of  the  missile,  (c)  The  resulting 
correlation  surface  with  the  position  of  the  correct  match  indicated  with  an  arrow. 


